Graph-Based Fusion of Imaging and Non-Imaging Data for Disease Trajectory Prediction

This study proposes a graph convolutional neural networks (GCN) architecture for fusion of radiological imaging and non-imaging tabular electronic health records (EHR) for the purpose of clinical event prediction. We focused on a cohort of hospitalized patients with positive RT-PCR test for COVID-19 and developed GCN based models to predict three dependent clinical events (discharge from hospital, admission into ICU, and mortality) using demographics, billing codes for procedures and diagnoses and chest X-rays. We hypothesized that the two-fold learning opportunity provided by the GCN is ideal for fusion of imaging information and tabular data as node and edge features, respectively. Our experiments indicate the validity of our hypothesis where GCN based predictive models outperform single modality and traditional fusion models. We compared the proposed models against two variations of imaging-based models, including DenseNet-121 architecture with learnable classification layers and Random Forest classifiers using disease severity score estimated by pre-trained convolutional neural network. GCN based model outperforms both imaging-only methods. We also validated our models on an external dataset where GCN showed valuable generalization capabilities. We noticed that edge-formation function can be adapted even after training the GCN model without limiting application scope of the model. Our models take advantage of this fact for generalization to external data. © 2023 IEEE.

Explainable AI for COVID-19 prognosis from early Chest X-ray and clinical data in the context of the COVID-CXR international hackathon

Although many AI-based scientific works regarding chest X-ray (CXR) interpretation focused on COVID-19 diagnosis, fewer papers focused on other relevant tasks, like severity estimation, deterioration, and prognosis. The same holds for explainable decisions to estimate COVID-19 prognosis as well. The international hackathon launched during Dubai Expo 2020, aimed at designing machine learning solutions to help physicians formulate COVID-19 patients' prognosis, was the occasion to develop a machine learning model capable of predicting such prognoses and justifying them through interpretable explanations. The large hackathon dataset comprised subjects characterized by their CXR and numerous clinical features collected during triage. To calculate the prognostic value, our model considered both patients' CXRs and clinical features. After automatic pre-processing to improve their quality, CXRs were processed by a Deep Learning model to estimate the lung compromise degree, which has been considered as an additional clinical feature. Original clinical parameters suffered from missing values that were adequately handled. We trained and evaluated multiple models to find the best one and fine-tune it before the inference process. Finally, we produced novel explanations, both visual and numerical, to justify the model predictions. Ultimately, our model processes a CXR and several clinical data to estimate a patient's prognosis related to the COVID-19 disease. It proved to be accurate and was ranked second in the final rankings with 75%, 73.9%, and 74.4% in sensitivity, specificity, and balanced accuracy, respectively. In terms of model explainability, it was ranked first since it was agreed to be the most interpretable by health professionals. © 2023 SPIE.

CRISIS: Cyber-Physical Social Distancing Based on Multi-Modal Data From Mobile Devices

Multi-modal sensors on mobile devices (e.g., smart watches and smartphones) have been widely used to ubiquitously perceive human mobility and body motions for understanding social interactions between people. This work investigates the correlations between the multi-modal data observed by mobile devices and social closeness among people along their trajectories. To close the gap between cyber-world data distances and physical-world social closeness, this work quantifies the cyber distances between multi-modal data. The human mobility traces and body motions are modeled as cyber signatures based on ambient Wi-Fi access points and accelerometer data observed by mobile devices that explicitly indicate the mobility similarity and movement similarity between people. To verify the merits of modeled cyber distances, we design the localization-free CybeR-physIcal Social dIStancing (CRISIS) system that detects if two persons are physically non-separate (i.e., not social distancing) due to close social interactions (e.g., taking similar mobility traces simultaneously or having a handshake with physical contact). Extensive experiments are conducted in two small-scale environments and a large-scale environment with different densities of Wi-Fi networks and diverse mobility and movement scenarios. The experimental results indicate that our approach is not affected by uncertain environmental conditions and human mobility with an overall detection accuracy of 98.41% in complex mobility scenarios. Furthermore, extensive statistical analysis based on 2-dimensional (2D) and 3-dimensional (3D) mobility datasets indicates that the proposed cyber distances are robust and well-synchronized with physical proximity levels. © 2002-2012 IEEE.

A Multimodal Framework for the Identification of Vaccine Critical Memes on Twitter

Memes can be a useful way to spread information because they are funny, easy to share, and can spread quickly and reach further than other forms. With increased interest in COVID-19 vaccines, vaccination-related memes have grown in number and reach. Memes analysis can be difficult because they use sarcasm and often require contextual understanding. Previous research has shown promising results but could be improved by capturing global and local representations within memes to model contextual information. Further, the limited public availability of annotated vaccine critical memes datasets limit our ability to design computational methods to help design targeted interventions and boost vaccine uptake. To address these gaps, we present VaxMeme, which consists of 10,244 manually labelled memes. With VaxMeme, we propose a new multimodal framework designed to improve the memes' representation by learning the global and local representations of memes. The improved memes' representations are then fed to an attentive representation learning module to capture contextual information for classification using an optimised loss function. Experimental results show that our framework outperformed state-of-the-art methods with an F1-Score of 84.2%. We further analyse the transferability and generalisability of our framework and show that understanding both modalities is important to identify vaccine critical memes on Twitter. Finally, we discuss how understanding memes can be useful in designing shareable vaccination promotion, myth debunking memes and monitoring their uptake on social media platforms. © 2023 ACM.

A Comparative Survey of Multimodal Multilabel Sentiment Analysis and Its Applications Initiated Due to the Impact of COVID-19

This study presents a detailed survey of different works related to sentiment analysis. The COVID-19 pandemic and its impact on people's mental health act as the driving force behind this survey. The survey can help study sentiment analysis and approaches taken in many studies to detect human emotions via advanced technology. It can also help in improving present systems by finding loopholes and increasing their accuracy. Various lexicon and ML-based systems and models like Word2Vec and LSTM were studied in the surveyed papers. Some of the current and future directions highlighted were Twitter sentiment analysis, review-based market analysis, determining changing behavior and emotions in a given time period, and detecting the mental health of employees, and students. This survey provides details related to trends and topics in sentiment analysis and an in-depth understanding of various technologies used in different studies. It also gives an insight into the wide variety of applications related to sentiment analysis. © 2022 IEEE.

Towards QoE Management for Post-Pandemic Online Learning : Invited Paper

During the Covid-19 pandemic, most schools had to adopt online learning, which is a special kind of e-learning that provides a virtual classroom via a live session for both teachers and learners. However, studies on education in the Covid-19 pandemic shows that there should be more efforts from researchers as well as governments to effectively support learners. In this paper, we focus on the problem of Quality of Experience in online learning. We discuss the enabling technologies of online learning. Also, we make an extensive review of QoE in video streaming, the key enabling technology of online leaning. Finally, the key challenges and potential solutions of QoE management for future online learning will be discussed. © 2022 IEEE.

Dockerized Knowledge-Oriented Multi-modal Social Event Detection System

Faced with an increasing amount of unstructured multimodal data appearing on various social platforms (e.g., Twitter/Instagram), we seek to effectively understand the complex social events portrayed on these platforms. However, conventional information extraction systems cannot understand these data because they cannot handle real-world analysis or require extensive tuning and many manually annotated examples to successfully comprehend these events. To solve this problem, this paper develops a knowledge-oriented artificial intelligence system that can identify and analyze these data and complex events and bring them to the user's attention. Our research aims to understand complex events described in multimedia inputs by developing a semi-automated system that identifies, links, and temporally sequences their subsidiary elements, the participants involved, as well as the complex event type. This project proposes a systematic analysis of world events, such as the Boston Marathon bombing, Capital Riots, Covid-19, etc. We have successfully evaluated our system on various datasets and have shown significant improvement compared to other previous methods. © 2022 IEEE.

Energy Demand Trends during COVID-19 Pandemic

COVID-19 pandemic lock-downs have led to the biggest fall in energy demand in over 70 years while also having an immense effect on the current energy mix. This study overviews the impacts of COVID-19 pandemic on the UK energy demand by analysing the associated electricity generation mix before and during COVID-19 pandemic. This analysis uses open-access data that is publicly available on the Official Carbon Intensity API for Great Britain. The scope of this paper is two-fold: first, to provide an overview of the lock-down measures in electricity demand and generation across the world, and second to identify the impact of lock-down restrictions on the British energy generation mix. It can be seen from the results that electricity generation by fossil fuels and renewable energy sources has shown opposite trends while the share of the later increased significantly during the lockdown period. © 2022 EUROSIS-ETI.

Modal analysis of novel coronavirus (SARS COV-2) using finite element methodology.

Many new engineering and scientific innovations have been proposed to date to passivate the novel coronavirus (SARS CoV-2), with the aim of curing the related disease that is now recognised as COVID-19. Currently, vaccine development remains the most reliable solution available. Efforts to provide solutions as alternatives to vaccinations are growing and include established control of behaviours such as self-isolation, social distancing, employing facial masks and use of antimicrobial surfaces. The work here proposes a novel engineering method employing the concept of resonant frequencies to denature SARS CoV-2. Specifically, "modal analysis" is used to computationally analyse the Eigenvalues and Eigenvectors i.e. frequencies and mode shapes to denature COVID-19. An average virion dimension of 63 nm with spike proteins number 6, 7 and 8 were examined, which revealed a natural frequency of a single virus in the range of 88-125 MHz. The information derived about the natural frequency of the virus through this study will open newer ways to exploit medical solutions to combat future pandemics.

Mortality Prediction Analysis among COVID-19 Inpatients Using Clinical Variables and Deep Learning Chest Radiography Imaging Features.

The emergence of the COVID-19 pandemic over a relatively brief interval illustrates the need for rapid data-driven approaches to facilitate clinical decision making. We examined a machine learning process to predict inpatient mortality among COVID-19 patients using clinical and chest radiographic data. Modeling was performed with a de-identified dataset of encounters prior to widespread vaccine availability. Non-imaging predictors included demographics, pre-admission clinical history, and past medical history variables. Imaging features were extracted from chest radiographs by applying a deep convolutional neural network with transfer learning. A multi-layer perceptron combining 64 deep learning features from chest radiographs with 98 patient clinical features was trained to predict mortality. The Local Interpretable Model-Agnostic Explanations (LIME) method was used to explain model predictions. Non-imaging data alone predicted mortality with an ROC-AUC of 0.87 ± 0.03 (mean ± SD), while the addition of imaging data improved prediction slightly (ROC-AUC: 0.91 ± 0.02). The application of LIME to the combined imaging and clinical model found HbA1c values to contribute the most to model prediction (17.1 ± 1.7%), while imaging contributed 8.8 ± 2.8%. Age, gender, and BMI contributed 8.7%, 8.2%, and 7.1%, respectively. Our findings demonstrate a viable explainable AI approach to quantify the contributions of imaging and clinical data to COVID mortality predictions.

Travel Mode Choice Analysis with Shared Mobility in Context of COVID-19

To analyze the impact of COVID-19 on the travel mode choice behavior with diverse shared mobility services, this study designed the stated preference (SP) questionnaire for the multi-modal transportation system which include conventional travel modes, ride hailing, ride sharing, car sharing, and bike sharing. The mixed Logit models with panel data were proposed to investigate the travel mode choices before and during COVID-19. The influence differences of explanatory variables are compared, and the joint effects of perceived pandemic severity and mode choice inertia are examined. Based on the elasticity analysis, the mode choice preferences are predicted corresponding to different management policies under COVID-19 pandemic. The results indicate that the perception to pandemic severity has significant impacts on the ridership of ride sharing and car sharing, and the mode choice inertia obviously affects the usage of ride hailing, car sharing, and bike sharing. When the perceived pandemic severity reduces to 30%~50%, the strategy of increasing parking charge to 1.6~3.0 times would reduce the usage of private car to pre-pandemic condition, and the car sharing with lower close contact risk could become a main substitute. When the perceived pandemic severity is higher than 60%, the strategy of increasing the travel safety of ride sharing to 1.4~3.6 times would improve the ridership. Copyright © 2022 by Science Press.

Multi-Modal Sarcasm Detection in Social Networks: A Comparative Review

Sentiment Analysis (SA) has become an extremely sought after area of research especially post COVID-19 when people used to spent a lot of time on the social media to interact with each other. This interaction was done through posts having both textual and visual cues and also by participating in online discussions forums. Some of the inherent challenges encountered in the process of SA include discernment of sarcasm, irony, humor, negation, multi-polarity or Aspect-Level Sentiment Analysis (ASA) etc. Researchers are now gradually shifting their focus to the identification and detection of sarcasm and how it can empower SA. Sarcasm expresses a person's downside feelings by using positive words in an implicit way. It also has an overall impact on increasing the efficiency of the SA models. Eliciting sarcastic statements is tough for humans as well as for machines without the knowledge of the context or background in which it is expressed, body language and/or facial expression of the speaker and his voice modulation. This review paper studies some of the approaches used for sarcasm detection and also guides researchers in exploring the different modalities of data for developing applications like a virtual chat-bot or assistant, depression analysis, stress management system at workplace etc. © 2022 IEEE.

COVID-19 Multi-Modal Data Analysis with Alexa Voice and Conversational AI Applications : Voice First System Tracking Novel Coronavirus

Novel Coronavirus (COVID-19) has changed the life of the planet. It is extremely important to monitor the situation in real time. Our secure methodology can help people to trace the situation in their country or state without touching a single computer key, just using voice-first computing devices. We have been using the data from multiple sources, created a suite of Alexa skills (Voice-first applications) and observed how the data evolves. The coronavirus data is downloaded automatically to the AWS Cloud and stored securely in the No-SQL DynamoDB database and S3 buckets to help users to monitor up-to-date statistics. Moreover, Alexa Echo devices with screens will display comprehensive graphs containing the most vital numbers - new cases, new deaths, mortality rate and hospitalizations since the pandemic started. Our system is safe, secure, automatic and resilient. It helps users to maintain social distancing and obtain up-to-date information about coronavirus in the location of interest without a single touch, just by using voice. During our journey we have designed and implemented many convenient commands, improving usability and multi-modal user experience. Our innovative approach, serverless architecture and Big Data methodology can help millions of people to stay on top of the coronavirus situation and make day-to-day choices using the information provided. It can also help officials to make educated decisions about opening certain businesses, institutions or activities. Since more and more voice assistants (AI devices) appear in public places - hotels, restaurants, and airports, our approach will help people to stay informed everywhere. Using our touch-free Alexa analytical skills will also promote social distancing. © 2021 IEEE.

MedSeq2Seq: A Medical Knowledge Enriched Sequence to Sequence Learning Model for COVID-19 Diagnosis

The COVID-19 pandemic has had a severe impact on humans' lives and and healthcare systems worldwide. How to early, fastly and accurately diagnose infected patients via multimodal learning is now a research focus. The central challenges in this task mainly lie on multi-modal data representation and multi-modal feature fusion. To solve such challenges, we propose a medical knowledge enriched multi-modal sequence to sequence learning model, termed MedSeq2Seq. The key components include two attention mechanisms, viz. intra-modal (Ia) and inter-model (Ie) attentions, and a medical knowledge augmentation mechanism. The former two mechanisms are to learn multi-modal refined representation, while the latter aims to incorporate external medical knowledge into the proposed model. The experimental results show the effectiveness of the proposed MedSeq2Seq framework over state-of-the-art baselines with a significant improvement of 1%-2%. © 2021 IEEE.

Multi-modal Information Fusion-powered Regional Covid-19 Epidemic Forecasting

With the current raging spread of the COVID19, early forecasting of the future epidemic trend is of great significance to public health security. The COVID-19 is virulent and spreads widely. An outbreak in one region often triggers the spread of others, and regions with relatively close association would show a strong correlation in the spread of the epidemic. In the real world, many factors affect the spread of the outbreak between regions. These factors exist in the form of multimodal data, such as the time-series data of the epidemic, the geographic relationship, and the strength of social contacts between regions. However, most of the current work only uses historical epidemic data or single-modal geographic location data to forecast the spread of the epidemic, ignoring the correlation and complementarity in multi-modal data and its impact on the disease spread between regions. In this paper, we propose a Multimodal InformatioN fusion COVID-19 Epidemic forecasting model (MINE). It fuses inter-regional and intra-regional multi-modal information to capture the temporal and spatial relevance of the COVID-19 spread in different regions. Extensive experimental results show that the proposed method achieves the best results compared to state-of-art methods on benchmark datasets. © 2021 IEEE.

SafeCampus: Multimodal-Based Campus-Wide Pandemic Forecasting

The motivation of this work is to build a multimodal-based COVID-19 pandemic forecasting platform for a large-scale academic institution to minimize the impact of COVID-19 after resuming academic activities. The design of this multimodality work is steered by video, audio, and tweets. Before conducting COVID-19 prediction, we first trained diverse models, including traditional machine learning models (e.g., Naive Bayes, support vector machine, and TF-IDF) and deep learning models [e.g., long short-term memory (LSTM), MobileNetV2, and SSD], to extract meaningful information from video, audio, and tweets by 1) detecting and counting face masks, 2) detecting and counting cough for potential infected cases, and 3) conducting sentiment analysis based on COVID-19-related tweets. Finally, we fed the multimodal analysis results together with daily confirmed cases data and social distancing metrics into the LSTM model to predict the daily increase rate of confirmed cases for the next week. Important observations with supporting evidence are presented. © 1997-2012 IEEE.